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About me 


٠ byears - Back, Front and everything in between 
٠ 3 years at NGINX 

٠ NGINX Amplify - Monitoring and Analytics 

٠ NGINX Controller - Amplify + Management 
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The problem 


* NGINX is a high performance, feature rich... 
o Web server 
o Load balancer 
o Reverse proxy 
o Cache 


* NGINX configs are powerful and complex. 
Sf fa irt i Fe PUR هی‎ 


C» Complicated 


The problem 


85 modules But there's more: 

e Conditionals (i f) 
214 directives 

٠ Go-Tos (rewrite) 


208 variables * Case statements (map and geo) 


The problem - Inheritance 


server ( 
listen 80; 
er X-Request-ID S$request. id 
always; 


location / ( 
return 200 “index”; 


) 
WEE oer 1 


add header Cache-Control 
cache" always; 
return 200 "other"; 


"no- 


GET / HTTP/1.0 


HTTP/1.1 200 ok 

Server: nginx/x.xx.x 

Date: Mon, 24 Sep 2018, 10:15:22 GMT 
Content-Type: application/octet-stream 
Content-Length: 5 

Connection: close 

X-Request-ID: 925d955df378b5e369df9a180669f3ca 


index 


GET /other/ HTTP/1.0 


HTTP/1.1 200 ok 

Server: nginx/Xx.xx.x 

Date: Mon, 24 Sep 2018, 10:15:22 GMT 
Content-Type: application/octet-stream 
Content-Length: 5 

Connection: close 


یی مه 


other 


The problem - Best practices 


server ( 
listen 80: proxy.set. header 
proxy.set. header 


: proxy.set. header 
B Ee proxy. set. header 
include proxy. headers.cont; 9 proxy.set. header 


proxy pass «some. host»; proxy.set header 


Host Shost; 

X-Real-IP $remote_addr; 
X-Forwarded-For Sremote_addr; 
X-Forwarded-Host Shost; 
X-Forwarded-Port S$server port; 
X-Forwarded-Protocol Sscheme; 


The problem - Bad practices 


$host - In this order of $http host - host name from the 


server ( precedence: host name from the "Host" request header field. 
. request line, host name from the 
listen 80; 


"Host" request header field, or the $http «name» - arbitrary request 
server name matching a request. 


. header field; name is the field 
location / ( name converted to lower case with 


include dashes replaced by underscores. 


proxy. headers.conf ; 
proxy.pass http:// 


dlock gauravphoenix$ curl http://13 41.6! atest/meta-data/iam 


security-credenti 


https://redlock.io/blog/instance-metadata-api-a-modern-day-trojan-hor 
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Were not the only ones... 


* There are some other open source tools which do some 
analysis of NGINX configs. 
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Parsing NGINX co 


nfigs 


Parsing 


* Before we can analyze a config, we need to be able to 
understand NGINX configs. 
س‎ = 


4 Parse - In linguistics, to divide language 
into small components that can be 
analyzed [1]. 


[1] https://www.webopedia.com/TERM/P/parse.html 


A brief history 


* NGINX itself does not have a config parser. 
٠ https://github.com/fatiherikli/nginxparser 


o Uses pyparsing to parse NGINX configs 
o Slow, susceptible to edge cases. 


٠ Paul McGuire optimized our usage of pyparsing for 
„59% performance improvement. 


٠ Still expensive and we continued to find edge cases two 
years later. [1] 


[1] https://github.com/nginxinc/nginx-amplify-agent/blob/484f20a902ed07dc4b50107c0ad6c5d7f14e4681/ 0 
44  amplify/agent/objects/nginx/config/parser.py 


©) Crossplane 


٠ So we revisit tooling...not much has changed. 
* NGINX Config to JSON (and back). 
* Faster and much more reliable. 


* Even uses NGINX directive definitions for 
validation. [1] 


٠ https://github.com/nginxinc/crossplane 


[1] https://github.com/nginxinc/crossplane/blob/master/crossplane/analyzer.py 0 


NGINX directive definitions 


gshulegaard@sandbox-conf-gsh:/etc/nginx$ sudo nginx -t 


nginx: configuration file /etc/nginx/nginx.conf test failed 


Files: 1 ^ 


/etc/nginx/nginx.conf 40 lines 1053 bytes Oct 2, 2018, 14:32:08 UTC-07 


simple.conf 


( 

۱ "status" :"ok", 
"errors" :[], 
"config': [ 


"file":"simple.conf", 
"status": "ok", 
"errors" <1], 


events) { 
. worker.connections 1024; 


A | 


http ( "parsed" :[ 
server { { 
à = "directive" :"events", 
-8080 : „args :[], 
127.0.0.1:8080; "line": 1, 


default_ser { 
location / { 

return 200 "foo "directive": 

i "args" :["'1024"], 
y "line" :2, 


y ) 


bar baz" 


171717 


"worker. connections" 


"directive" 
"args" :[], 
"line":5 
"Brook" 


"http", 


"directive" 
"args" :[], 
"line" :6 
"block": | 


: "server", 


"directive" 
"args": 


: "listen", 


[*127.0.0.1:8089], j 


"directive" 


1 


"server name" 
'args" 


["default server' du 


"directive":" 


bar baz"], 


"line":8, 

}, 

را 
directive”‏ 
"args" :["/"],‏ 
"line" :9‏ 
Ê‏ : ا 


return", 


:" location", 


A few examples 


Inheritance 


server ( ۳ 
en 80; block: [ 
و‎ add head X-Request-ID Srequest_id 
always, ENE. ler”, وله‎ 


location / ( 


return 200 "index"; location", 


) her/“], 
= der Cache-Control “no-cache” 
always; 
return 200 “other”; ) ] 
) 1 


Inheritance 


٠ Initialize a flag (False). 


٠ Whenever you find an 
add header directive 


consult the flag. 

o If the flag is False, flip it and 
continue. 

o If the flag is already True, 
you've found an inheritance 
rewrite. 
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5 False 
block: [ 
{“directive”: "add header", .), True 
( 
"directive": "location", 


“args”: ["/other/"], 


an ee : “a له‎ Warm 
] 
Ja 


Best practices 


Server 4 e Initialize a “context”. 


listen 80; 


mE * When you find 
ocation / ( 
include proxy. headers.conf; proxy ser heade E 
proxy_pass <some_host>; = — 
overwrite the context. 


٠ When you find 
proxy pass, consult the 


context. 


Ô 
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Bad practices 


server ( 
listen 80; 


location / ( 


ca epee or 
; _pass http://Shttp. host; 


) 
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3 


و cmm‏ وا 
"args": : http host"],‏ 


Bad practices 


٠ Lookforproxy pass 5 

directives. ) تت‎ ۱ 
. Check the first arg... an 
e Split by "/" and check if the : 

first variable or the first 

variable after scheme is 

vulnerable. 

o http://Shttp_host/ % 

O os 

o http://10-10.10.54/?arg=Shttp_host eff 
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0 Controller 


Systems 


€ AWS_EC2_Instance_2 


€ AWS_EC2_Instance_1 


NGINX Controller 


= Overview Configure Graphs Dashboards v Alerts A 


nginx-plus-114-p1 07۰ v 
Q 9 p UTC-07 * 11:28 


€- Back 


Config analysis 


Parser errors: O 

Files: 1 v 

Servers: 2 

Locations: 5 total (3 prefix, 2 exact, O regex) 
Upstreams: 0 total 


Recommendations (Errors: 0, Warnings: 2) 


Missing default server 

Default server is the virtual server that will process requests that can't be matched against any of the configured server name. If 
none of the listen directives have the default server parameter, the first server with a given IP/port combination becomes the 
default one 


Add the default server parameter for each unique listen. For instance, if you have several servers listening on port 80, add 


default server to the one that should be used for all requests that don't match any configured server name 


Check the following files: 
c/nginx/nginx.conf, line 117 


/etc/nginx/nginx.conf, line 100 


Missing server. name in a virtual server configuration 
When nginx accepts a request, the first step is to determine which virtual server the request should be routed to. Without a 
server. name directive set for a virtual server, the request may hit the wrong one 


Always configure a unique server. name directive for a server. Make sure the server. name contains all possible site names 
+ New Instance 
Check the following file: 
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/etc/nginx/nginx.conf, line 99 


(0) 


Currently available reports 
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Proxy pass headers 
Missing server names 
Worker processes 
Missing default conf 
Proxy buffering 

Proxy request buffering 
Missing default server 
Regex checks 

Stub status ACL 

Plus status ACL 

Plus API ACL 

Dangerous rewrite checks 
Wildcard TCP socket overlaps 


FastCGl params 

Listen overlaps 

Obsolete SSL configuration 
Missing listens 

Missing error log 

Multiple stub status 

SSL protocol checks 
Missing SSL protocol 

SSL cipher checks 
Insecure proxy pass 
Header inheritance overlap 


Debug connection without — 
with-debug 


Configuration 
overview 


SSL certificate 
expiration 


SSL certificate info 
checks 


NGINX security 
advisories 


events { 
worker_connections 1024; 


} 


http { 

log. format main ext '$remote addr - $remote_user [$time local] "$request" 
'$status $body bytes sent "$http referer" 
'"$http user agent" "$http. x forwarded for" 
'"$host" sn-"$server name" ' 
'rid-$http x request id rt-$request time 
'ua-"$upstream addr" us-"$upstream status" 
'ut-"$upstream response time" ul-"$upstream response length" 
'cs-$upstream cache. status'; 


06655-109 /var/log/nginx/access.log main_ext; 


error log /var/log/nginx/error.log warn; 


server { 
listen 80; 
add_header X-Request-ID $request_id always; 


location / 1 
proxy. pass http://127.0.0.1:8080; 
} 


location /stub_status { 
stub_status on; 


Overview Graphs Dashboards v Alerts A Grant Hulegaard v 


nginx-plus-r16 UTC-07 *15:06 v 


e sandbox-conf-gsh Build Static analysis Virtual servers 
Host sandbox-conf-gsh Updated Oct 2, 2018, 15:05:40 UTC-07 Servers 1 
Version nginx-plus-r16 Files 1 Server names 0 
Status Up Locations 2 Total IPv4 addresses 0 
Upstreams 0 Listen IP/ports 1 
Security advisories SSL 
Major severity 0 Certificates 0 
Medium severity 0 Expired 0 
Minor severity 0 Warnings 0 
SSL/TLS servers 0 
HTTP/2 servers 0 
Library OpenSSL1.0.1f6 Jan 2014 


+ New System 


Missing server name in a virtual server configuration 


When nginx accepts a request, the first step is to determine which virtual server the request should be 
routed to. Without a server name directive set for a virtual server, the request may hit the wrong one. 


Always configure a unique server. name directive for a server. Make sure the server name contains all 
possible site names. 


Missing ACL for stub status 


No access control list configured for nginx stub status. Anyone knowing or guessing the stub, status 
location can see your server counters. This may lead to an undesired information disclosure. 


Check the following file: 
/etc/nginx/nginx.conf, line 18 


Add an ACL to the stub, status configuration. If you don't need external access to status counters, change 
the virtual server listen configuration to only use the loopback interface. 


Missing HTTP header definitions in proxy. pass Example (very strict) configuration is: 
When nginx proxies a request, the HTTP headers passed to the application may change unless explicitly 
configured. For example, the Host header is set by default to the value of the 5proxy host runtime Be 


listen 127.0.0.1:80; 
server name 127.0.0.1; 


variable. Without a clear definition of the headers the application behavior may be different from what you 


expect. 
location /nginx status ( 


stub status on; 
allow 127.0.0.1; 
deny all; 


Best practice is to configure a clear set of headers with proxy. pass. The Host header is always important. 
Add the following to your nginx configuration: 


proxy set header Host Shost; 


and optionally the following: Check the following file: 
/etc/nginx/nginx.conf, line 27 


proxy set header X-Real-IP $remote addr; 
proxy set header X-Forwarded-For $remote_addr; 
proxy set header X-Forwarded-Host $host; 
proxy set header X-Forwarded-Port $server port; 


proxy set header X-Forwarded-Protocol Sscheme; 


Check the following file: 
/etc/nginx/nginx.conf, line 23 


Wrap-up © 


Wrap-up 


* Talked about why you would want to 
analyze NGINX configs. 


٠ Parsed configs with crossplane. 
٠ Walked through some example analyses. 


٠ Talked about analysis in NGINX Controller 
today. 


* Used the analyzer with a live example. 


NGINX — 


- Thank you 


grant.hulegaard@nginx.com 
@gshulegaard 


